Displaying an object indicator

ABSTRACT

Examples disclosed herein describe, among other things, a computing system. The computing system may in some examples include a touch-sensitive surface, a display, and at least one camera to capture an image representing an object disposed between the camera and the touch-sensitive surface. The computing system may also include a detection engine to determine, based at least on the image, display coordinates, where the display coordinates may correspond to the object&#39;s projection onto the touch-sensitive surface, and the display is not parallel to the touch-sensitive surface. In some examples, the detection engine is also to display an object indicator at the determined display coordinates on the display.

PRIORITY INFORMATION

This application is a continuation of U.S. National Stage applicationSer. No. 15/514,639 filed on Mar. 27, 2017, which claims priority toInternational Application No. PCT/US2014/058189 filed on Sep. 30, 2014.The contents of which are incorporated herein by reference in itsentirety.

BACKGROUND

Many computing systems today include a display, a camera, and an inputdevice. In some systems, the display may be a touch-sensitive display,sometimes referred to as a touch screen. Input devices include, forexample, a mouse, a keyboard, or a touch-sensitive surface capable ofdetecting physical objects that come into contact therewith.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description references the drawings, wherein:

FIG. 1 is a schematic perspective view of an example computing systemcomprising an identification engine;

FIG. 2 is another schematic perspective view of the example computingsystem of FIG. 1;

FIG. 3 is a schematic side view of the example computing system of FIG.1;

FIG. 4 is a schematic front view of the example computing system of FIG.1;

FIG. 5 is a schematic side view of the example computing system of FIG.1 during an example operation;

FIG. 6 is a schematic front view of the example computing system of FIG.1 during another example operation;

FIG. 7A is another side view of the example computing system of FIG. 1;

FIG. 7B is a perspective view of the example computing system of FIG.7A;

FIG. 8 is a block diagram of an example computing device of the examplecomputing system of FIG. 1;

FIG. 9 is a block diagram of another example computing device of theexample computing system of FIG. 1; and

FIG. 10 is a flowchart of an example method for displaying at least oneobject indicator.

DETAILED DESCRIPTION

In some computing systems, user experience may be enhanced by allowingthe user to use, in conjunction with an application running on thecomputing system, objects such as a stylus, a fingertip or fingertips, agame piece, and the like. The objects may be used as input devices,application (e.g., gaming) accessories, or for any other purposes. Whenusing such objects, it may be difficult for the user to know whether andwhere the computing system has detected the object(s) if the user doesnot receive appropriate indication by the computing system.

In some examples described herein, a computing system is disclosed. Thecomputing system may include, for example, a surface (e.g., atouch-sensitive surface), a display, and at least one camera to capturean image representing an object disposed, for example, between thecamera and the surface. The system may also include a detection engineto determine, based at least on the image, display coordinatescorresponding to the objects projection onto the touch-sensitivesurface, where the display may not be parallel to the touch-sensitivesurface. The system may also select an object indicator from at leasttwo different object indicators based at least on whether the object istouching the touch-sensitive surface, and to provide the selected objectindicator to the display for displaying the selected object indicator atthe determined display coordinates on the display.

Referring now to the drawings, FIGS. 1-6 are schematic views of anexample computing system 100 comprising a detection engine 170. In theexamples of FIGS. 1-6, system 100 may include a support structure 110, acomputing device 150, a display 152, and a touch-sensitive surface 200.System 100 may also include a sensor bundle 164 pointed, for example, attouch-sensitive surface, to capture one or more images representing anobject disposed on or above touch sensitive surface 200. Computingdevice 150 may include a detection engine 170 to determine, based atleast on one or more images (received, for example, from sensor bundle164) display coordinates associated with coordinates of the object'sprojection onto touch-sensitive surface, and to display an objectindicator at the determined display coordinates on the display.

Computing device 150 may comprise any suitable computing devicecomplying with the principles disclosed herein. As used herein, a“computing device” may comprise an electronic display device, asmartphone, a tablet, a chip set, an all-in-one computer (e.g., a devicecomprising a display device that also houses processing resource(s) ofthe computer), a desktop computer, a notebook computer, workstation,server, any other processing device or equipment, or a combinationthereof. In this example, device 150 is an all-in-one computer having acentral axis or center line 155, first or top side 150A, a second orbottom side 150B axially opposite the top side 150A, a front side 150Cextending axially between sides 150A and 150B, a rear side 150D alsoextending axially between sides 150A and 150B and generally radiallyopposite front side 150C. A display 152 is disposed along front side150C and defines a viewing surface of computing system 100 to displayimages for viewing by a user of system 100. In examples describedherein, a display may include components of any technology suitable fordisplaying images, video, or the like.

In some examples, display 152 may be a touch-sensitive display. Inexamples described herein, a touch-sensitive display may include, forexample, any suitable technology (e.g., components) for displayingimages, video, or the like, and may include any suitable technology(e.g., components) for detecting physical contact (e.g., touch input),such as, for example, a resistive, capacitive, surface acoustic wave,infrared (IR), strain gauge, optical imaging, acoustic pulserecognition, dispersive signal sensing, or in-cell system, or the like.In examples described herein, display 152 may be referred to as atouch-sensitive display 152. Device 150 may further include a camera154, which may be a web camera, for example. In some examples, camera154 may capture images of a user positioned in front of display 152. Insome examples, device 150 may also include a microphone or other deviceto receive sound input (e.g., voice input from a user).

In the example of FIGS. 1-6, support structure 110 includes a base 120,an upright member 140, and a top 160. Base 120 includes a first or frontend 120A, and a second or rear end 120B. Base 120 may engage with asupport surface 15 to support the weight of at least a portion of thecomponents of system 100 (e.g., member 140, unit 180, device 150, top160, etc.). In some examples, base 120 may engage with support surface15 in this manner when system 100 is configured for operation. In theexample of FIGS. 1-6, front end 120A of base 120 includes a raisedportion 122 that may be disposed above and separated from supportsurface 15 (creating a space or clearance between portion 122 andsurface 15) when base 120 is disposed on support surface 15 asillustrated in FIG. 2, for example. In such examples, a portion of aside of touch-sensitive surface 200 may be disposed in (e.g., receivedwithin) the space formed between portion 122 and surface 15. In suchexamples, placing a portion of surface 200 within the space created byportion 122 and surface 15 may assist with the proper alignment ofsurface 200. In other examples, other suitable methods or devices may beused to assist with the alignment of surface 200.

Upright member 140 includes a first or upper end 140A, a second or lowerend 140B opposite the upper end 140A, a first or front side 140Cextending between the ends 140A and 140B, and a second or rear side 140Dopposite the front side 140C and also extending between the ends 140Aand 140B. Lower end 140B of member 140 is coupled to rear end 120B ofbase 120, such that member 140 extends substantially upward from supportsurface 15.

Top 160 includes a first or proximate end 160A, a second or distal end160B opposite the proximate end 160A, a top surface 160C extendingbetween ends 160A and 160B, and a bottom surface 160D opposite the topsurface 160C and also extending between ends 160A and 160B. Proximateend 160A of top 160 is coupled to upper end 140A of upright member 140such that distal end 160B extends outward from upper end 140A of uprightmember 140. As such, in the example shown in FIG. 2, top 160 issupported at end 160A (and not at end 160B), and may be referred toherein as a cantilevered top. In some examples, base 120, member 140,and top 160 may be monolithically formed. In other examples, two or moreof base 120, member 140, and top 160 may be formed of separate pieces(i.e., not monolithically formed).

Touch-sensitive surface 200 may include a central axis or centerline205, a first or front side 200A, and a second or rear side 200B axiallyopposite the front side 200A. Touch-sensitive surface 200 may compriseany suitable technology for detecting physical contact with surface 200by an object such as hand or other objects (e.g., objects containingconductive material) whose placement on or close to surface 200 maycause a detectible change in capacitance or other parameters of surface200. For example, touch-sensitive surface 200 may comprise any suitabletechnology for detecting (and in some examples tracking) one or multipletouch inputs by a user to enable the user to interact, via such touchinput, with software being executed by device 150 or another computingdevice. As another example, touch-sensitive surface 200 may comprise anysuitable technology for detecting (and in some examples tracking) one ormultiple objects disposed on touch-sensitive surface 200 to enable theuser to interact, via placement, rotation, movement, and othermanipulations of such object(s), with software being executed by device150 or another computing device.

In examples described herein, touch-sensitive surface 200 may be anysuitable touch-sensitive planar (or substantially planar) object, suchas a touch-sensitive mat, tabletop, sheet, etc. In some examples,touch-sensitive surface 200 may be disposed horizontally (orapproximately or substantially horizontally). For example, surface 200may be disposed on support surface 15, which may be horizontal (orapproximately or substantially horizontal).

In some examples, all or substantially all of surface 200 may be capableof detecting touch input as described above. In other examples, lessthan all of surface 200 may be capable of detecting touch input asdescribed above. For example, surface 200 may comprise a touch-sensitiveregion 202, extending over less than all of surface 200, wherein region202 is capable of detecting touch input as described above. In otherexamples, region 202 may extend over substantially all of surface 200(e.g., may be substantially coterminous with surface 200). Region 202may be substantially aligned with axis 205.

As described above, surface 200 may be aligned with base 120 ofstructure 110 to assist with proper alignment of surface 200 (e.g., atleast during operation of system 100). In the example of FIGS. 1-6, rearside 200B of surface 200 may be disposed between raised portion 122 ofbase 120 and support surface 15, such that rear end 200B is aligned withfront side 120A of base 120 to assist with proper overall alignment ofsurface 200 (and particularly proper alignment of region 202) with othercomponents of system 100. In some examples, surface 200 may be alignedwith device 150 such that the center line 155 of device 150 issubstantially aligned with center line 205 of surface 200. In otherexamples, surface 200 may be differently aligned with device 150.

In some examples, surface 200 and device 150 may be communicativelyconnected (e.g., electrically coupled) to one another such that userinputs received by surface 200 may be communicated to device 150.Surface 200 and device 150 may communicate with one another via anysuitable wired or wireless communication technology or mechanism, suchas, for example, WI-FI, BLUETOOTH, ultrasonic technology, electricalcables, electrical leads, electrical conductors, electricalspring-loaded pogo pins with magnetic holding force, or the like, or acombination thereof. In the example of FIGS. 1-6, exposed electricalcontacts disposed on rear side 200B of surface 200 may engage withcorresponding electrical pogo-pin leads within portion 122 of base 120to communicate information (e.g., transfer signals) between device 150and surface 200 during operation of system 100. In such examples, theelectrical contacts may be held together by adjacent magnets (located inthe clearance between portion 122 of base 120 and surface 15) tomagnetically attract and hold (e.g., mechanically) a correspondingferrous and/or magnetic material disposed along rear side 200B ofsurface 200.

Referring to FIG. 3, projector unit 180 comprises an outer housing 182,and a projector assembly 184 disposed within housing 182. Housing 182includes a first or upper end 182A, a second or lower end 182B oppositethe upper end 182A, and an inner cavity 183. In the example of FIG. 3,housing 182 further includes a coupling or mounting member 186 to engagewith and support device 150 (e.g., at least during operation of system100). Member 186 may be any suitable mechanism or device for suspendingand supporting any suitable computing device 150 as described herein.For example, member 186 may comprise a hinge that includes an axis ofrotation such that device 150 may be rotated (e.g., by a user) about theaxis of rotation to attain a desired angle for viewing display 152. Insome examples, device 150 may permanently or semi-permanently attachedto housing 182 of unit 180. In some examples, housing 180 and device 150may be integrally or monolithically formed as a single unit.

Referring to FIG. 4, in some examples, when device 150 is suspended fromstructure 110 via mounting member 186 on housing 182, projector unit 180(i.e., both housing 182 and assembly 184) may be substantially hiddenbehind device 150 when system 100 is viewed from the front (i.e.,substantially facing display 152 disposed on front side 150C of device150). In addition, as shown in FIG. 4, when device 150 is suspended fromstructure 110 as described above, projector unit 180 (i.e., both housing182 and assembly 184) and any image projected thereby may besubstantially aligned or centered with respect to center line 155 ofdevice 150.

Referring again to FIG. 3, projector assembly 184 is disposed withincavity 183 of housing 182, and includes a first or upper end 184A, asecond or lower end 184B opposite the upper end 184A. Upper end 184A isproximate upper end 182A of housing 182 while lower end 184B isproximate lower end 182B of housing 182. Projector assembly 184 maycomprise any suitable digital light projector assembly for receivingdata from a computing device (e.g., device 150) and projecting image(s)(e.g., out of upper end 184A) that correspond with that input data. Forexample, in some implementations, projector assembly 184 may comprise adigital light processing (DLP) projector or a liquid crystal on silicon(LCoS) projector which are advantageously compact and power efficientprojection engines capable of multiple display resolutions and sizes,such as, for example, standard XGA resolution (1024×768 pixels) with a4:3 aspect ratio, or standard WXGA resolution (1280×800 pixels) with a16:10 aspect ratio. Projector assembly 184 is further communicativelyconnected (e.g., electrically coupled) to device 150 in order to receivedata therefrom and to produce (e.g., project) light and image(s) fromend 184A based on the received data. Projector assembly 184 may becommunicatively connected to device 150 via any suitable type ofelectrical coupling, for example, or any other suitable communicationtechnology or mechanism described herein. In some examples, assembly 184may be communicatively connected to device 150 via electricalconductor(s), WI-FI, BLUETOOTH, an optical connection, an ultrasonicconnection, or a combination thereof. In the example of FIGS. 1-6,device 150 is communicatively connected to assembly 184 throughelectrical leads or conductors (e.g., as described above in relation tosurface 200 and base 120) disposed within mounting member 186 such that,when device 150 is suspended from structure 110 through member 186, theelectrical leads disposed within member 186 contact corresponding leadsor conductors disposed on device 150.

Referring still to FIG. 3, top 160 further includes a fold mirror 162and a sensor bundle 164. Mirror 162 includes a highly reflective surface162A that is disposed along bottom surface 160D of top 160 and ispositioned to reflect light, image(s), etc., projected from upper end184A of projector assembly 184 toward surface 200 during operation.Mirror 162 may comprise any suitable type of mirror or reflectivesurface. In the example of FIGS. 1-6, fold mirror 162 may comprise astandard front surface vacuum metalized aluminum coated glass mirrorthat acts to fold light emitted from assembly 184 down to surface 200.In other examples, mirror 162 may have a complex aspherical curvature toact as a reflective lens element to provide additional focusing power oroptical correction.

Sensor bundle 164 includes at least one sensor (e.g., camera, or othertype of sensor) to detect, measure, or otherwise acquire data based onthe state of (e.g., activities occurring in) a region between sensorbundle 164 and surface 200. The state of the region between sensorbundle 164 and surface 200 may include object(s) disposed on or abovesurface 200, or activities occurring on or near surface 200. In theexample of FIG. 3, bundle 164 includes an RGB camera (or image sensor)164A, an IR camera (or IR sensor) 164B, a depth camera (or depth sensor)164C, and an ambient light sensor 164D. In examples described herein, acamera may be referred to as a “sensor”.

In some examples, RGB camera 164A may be a camera to capture colorimages (e.g., at least one of still images and video). In some examples,RGB camera 164A may be a camera to capture images according to the RGBcolor model, which may be referred to herein as “RGB images”. It isappreciated, however, that in other examples, RGB camera 164A may be acamera to capture image according to other color models, such as YUV,YCbCr, RAW, and so forth. In some examples, RGB camera 164A may captureimages with relatively high resolution, such as a resolution on theorder of multiple megapixels (MPs), for example. As an example, RGBcamera 164A may capture color (e.g., RGB) images with a resolution of 14MPs. In other examples, RBG camera 164A may capture images with adifferent resolution. In some examples, RGB camera 164A may be pointedtoward surface 200 and may capture image(s) of surface 200, object(s)disposed between surface 200 and RGB camera 164A (e.g., hovering abovesurface 200 or touching surface 200), or a combination thereof.

IR camera 164B may be a camera to detect intensity of IR light at aplurality of points in the field of view of the camera 164B. In examplesdescribed herein, IR camera 164B may operate in conjunction with an IRlight projector 166 of system 100 to capture IR images. In suchexamples, each IR image may comprise a plurality of pixels eachrepresenting an intensity of IR light detected at a point represented bythe pixel. In some examples, top 160 of system 100 may include an IRlight projector 166 to project IR light 167 toward surface 200 and IRcamera 164B may be pointed toward surface 200. In such examples, IRcamera 164B may detect the intensity of IR light reflected by surface200, object(s) disposed between surface 200 and IR camera 164B (e.g.,hovering above surface 200 or touching surface 200), or a combinationthereof. In some examples, IR camera 164B may exclusively detect IRlight 167 projected by IR light projector 166 (e.g., as reflected fromsurface 200, object(s), etc., or received directly from an infraredsource).

Depth camera 164C may be a camera (sensor(s), etc.) to detect therespective distance(s) (or depth(s)) of portions of object(s) in thefield of view of depth camera 164C. As used herein, the data detected bya depth camera may be referred to herein as “distance” or “depth” data.In examples described herein, depth camera 164C may capture amulti-pixel depth image (e.g., a depth map), wherein the data of eachpixel represents the distance or depth (measured from camera 164C) of aportion of an object at a point represented by the pixel. Depth camera164C may be implemented using any suitable technology, such asstereovision camera(s), a single IR camera sensor with a uniform floodof IR light, a dual IR camera sensor with a uniform flood of IR light,structured light depth sensor technology, time-of-flight (TOF) depthsensor technology, or a combination thereof. In some examples, depthsensor 164C may indicate when an object (e.g., a three-dimensionalobject) is on surface 200. In some examples, depth sensor 164C maydetect at least one of the presence, shape, contours, motion, and therespective distance(s) of an object (or portions thereof) placed onsurface 200 or hovering above surface 200.

Ambient light sensor 164D may be arranged to measure the intensity oflight in the environment surrounding system 100. In some examples,system 100 may use the measurements of sensor 164D to adjust othercomponents of system 100, such as, for example, exposure settings ofsensors or cameras of system 100 (e.g., cameras 164A-164C), theintensity of the light emitted from light sources of system 100 (e.g.,projector assembly 184, display 152, etc.), or the like.

In some examples, sensor bundle 164 may omit at least one of sensors164A-164D. In other examples, sensor bundle 164 may comprise othercamera(s), sensor(s), or the like in addition to sensors 164A-164D, orin lieu of at least one of sensors 164A-164D. For example, sensor bundle164 may include a user interface sensor comprising any suitabledevice(s) (e.g., sensor(s), camera(s)) for tracking a user input devicesuch as, for example, a hand, stylus, pointing device, etc. In someexamples, the user interface sensor may include a pair of cameras whichare arranged to stereoscopically track the location of a user inputdevice (e.g., a stylus) as it is moved by a user about the surface 200(e.g., about region 202 of surface 200). In other examples, the userinterface sensor may additionally or alternatively include IR camera(s)or sensor(s) arranged to detect infrared light that is either emitted orreflected by a user input device. In some examples, sensor bundle 164may include a gesture camera to detect the performance of predefinedgestures by object(s) (e.g., hands, etc.). In some examples, the gesturecamera may comprise a depth camera and additional functionality todetect, track, etc., different types of motion over time.

In examples described herein, each of sensors 164A-164D of bundle 164 iscommunicatively connected (e.g., coupled) to device 150 such that datagenerated within bundle 164 (e.g., images captured by the cameras) maybe provided to device 150, and device 150 may provide commands to thesensor(s) and camera(s) of sensor bundle 164. Sensors 164A-164D ofbundle 164 may be communicatively connected to device 150 via anysuitable wired or wireless communication technology or mechanism,examples of which are described above. In the example of FIGS. 1-6,electrical conductors may be routed from bundle 164, through top 160,upright member 140, and projector unit 180 and into device 150 throughleads that are disposed within mounting member 186 (as described above).

Referring to FIGS. 5 and 6, during operation of system 100, projectorassembly 184 may project visible light 187 to reflect off of mirror 162towards surface 200 to thereby display visible image(s) on a projectordisplay space 188 of surface 200. In the example of FIGS. 5 and 6, space188 may be substantially rectangular, having a length 188L and a width188W. In some examples, length 188L may be approximately 16 inches,while width 188W may be approximately 12 inches. In other examples,length 188L and width 188W may have different values.

In some examples, cameras of sensor bundle 164 (e.g., cameras 164A-164C)are arranged within system 100 such that the field of view of each ofthe cameras includes a space 168 of surface 200 that may overlap withsome or all of display space 188, or may be coterminous with displayspace 188. In examples described herein, the field of view of thecameras of sensor bundle 164 (e.g., cameras 164A-164C) may be said toinclude space 168, though at times surface 200 may be at least partiallyoccluded by object(s) on or above surface 200. In such examples, theobject(s) on or over surface 200 may be in the field of view of at leastone of cameras 164A-164C. In such examples, sensors of sensor bundle 164may acquire data based on the state of (e.g., activities occurring in,object(s) disposed in) a region between sensor bundle 164 and space 168of surface 200. In some examples, both space 188 and space 168 coincideor correspond with region 202 of surface 200 such that functionalitiesof touch-sensitive region 202, projector assembly 184, and sensor bundle164 are all performed in relation to the same defined area. A field ofview 165 of the cameras of sensor bundle 164 (e.g., cameras 164A-164C)is schematically illustrated in FIG. 7A. In some examples, each of thecameras of sensor bundle 164 (e.g., cameras 164A-164C) may have adifferent field of view.

Referring now to FIGS. 5 and 6, device 150 may direct projector assembly184 to project image(s) onto surface 200 (e.g., onto region 202). Device150 may also display image(s) on display 152 (which may be the same asor different than the image(s) projected onto region 202 by projectorassembly 184). The image(s) projected by assembly 184 may compriseinformation and/or images produced by software being executed by device150. In some examples, a user may interact with the image(s) projectedon surface 200 and displayed on display 152 by physically engagingtouch-sensitive surface 200 in any suitable manner, such as with user'shand 35 (e.g., via touches, taps, gestures, or other touch input), witha stylus 25, or via any other suitable user input device(s). Asdescribed above, touch-sensitive surface 200 may detect such interactionvia physical engagement with surface 200. Also, in some examples,projector assembly 184 may also project image(s) (at least partially) onobjects disposed over surface 200 (e.g., hand 35, as shown in FIG. 5).

As an example, when a user interacts with touch-sensitive surface 200via physical contact, surface 200 may generate touch input informationand provide it to device 150 through any suitable connection (examplesof which are described above). In some examples, the OS may pass thereceived touch input to another application (e.g., program, etc.)executing on device 150. In response, the executing OS or applicationmay alter image(s) projected by projector assembly 184, image(s)displayed on display 152, or a combination thereof. As used herein, an“application”, “computer application”, or “service” may be a collectionof machine-readable instructions that are executable by a processingresource. In some examples, a user may similarly interact with image(s)displayed on display 152 (which may be a touch-sensitive display), orany other input device of device 150 (e.g., a keyboard, mouse, etc.).

In some examples, sensors (e.g., cameras) of sensor bundle 164 may alsogenerate system input which may be provided to device 150 for furtherprocessing. For example, system 100 may utilize camera(s) of bundle 164to detect at least one of the presence and location of an object (e.g.,user's hand 35, fingertip 37, stylus 25, etc.), and provide system inputinformation representing the detected information to device 150. In someexamples, system 100 may utilize one or more cameras to determine athree-dimensional location of an object, and provide that locationinformation to device 150. In some examples, system 100 may use at leasttwo images obtained from at least two different cameras of bundle 164(e.g., any combination of two cameras from cameras 164A, 164B, and 164C)to determine the three-dimensional location of the object. For example,at least two cameras of sensor bundle 164 may arranged to performstereoscopic object tracking of the object. In some examples, an object(e.g., stylus 25) may include at least one portion (e.g., a tip 26)coated with an infrared retro-reflective coating (e.g., paint) that mayserve as an infrared retro-reflector. In such examples, bundle 164 mayinclude IR camera(s) (or sensor(s)), as described above, which maydetect IR light that is reflected off the coated portion to enabledevice 150 to track the location of the coated portion of the object asit moves across region 202. In some examples, surface 200 (with image(s)projected on it by assembly 184) may serve as a second or alternativetouch-sensitive display within system 100. In addition, detection ofinteraction with image(s) displayed on surface 200 may be enhancedthrough use of sensors of sensor bundle 164 as described above.

In some examples, system 100 may capture two-dimensional (2D) image(s)or create a three-dimensional (3D) scan of a physical object such thatan image of the object may then be projected onto surface 200 forfurther use and manipulation thereof. For example, as shown in FIG. 6,an object 40 may be placed on surface 200 such that sensors of bundle164 (e.g., at least one of cameras 164A-164C) may detect at least one ofthe location, dimensions, and color of object 40, to enhance the 2Dimage(s) or create the 3D scan thereof. In such examples, theinformation gathered by the sensors of bundle 164 may be provided todevice 150 (e.g., an OS, application, service, etc., of device 150), asdescribed above. In some examples, after receiving the information,device 150 (e.g., the OS, application, service, etc.) may directprojector assembly 184 to project an image of object 40 onto surface200. Object 40 may be, for example, hand 35, fingertip 37, stylus 25, orany other physical object, such as a game piece, a book, a mug, a pen, adocument, a photo, and the like.

FIGS. 7A and 7B illustrate a side view and a perspective view,respectively, of an example computing system 100 comprising detectionengine 170. In the example of FIGS. 7A and 7B, the user may usefingertips 37 a-37 e of a left hand 35L and fingertips 37 f-37 j asobjects that may be detected and processed by detection engine 170 asfurther described in detail below.

FIG. 8 is a block diagram of an example portion of computing system 100of FIG. 1 comprising detection engine 170. In particular, FIG. 8illustrates an example of computing device 150 that comprises detectionengine 170 and a computer-readable medium 320, and is communicativelyconnected to at least one camera (e.g., camera 164A) of sensor bundle164 (as described above), to touch-sensitive surface 200, and to display152, as described above. Although not shown in FIG. 8 computing device150 may also be communicatively connected to other components of system100, as described above.

Computing device 150 (or any other computing device implementingdetection engine 170) may include at least one processing resource. Inexamples described herein, a processing resource may include, forexample, one processor or multiple processors included in a singlecomputing device or distributed across multiple computing devices. Asused herein, a “processor” may be at least one of a central processingunit (CPU), a semiconductor-based microprocessor, a graphics processingunit (GPU), a field-programmable gate array (FPGA) configured toretrieve and execute instructions, other electronic circuitry suitablefor the retrieval and execution instructions stored on amachine-readable storage medium, or a combination thereof.

As noted above, computing device 150 may comprise detection engine 170.In some examples, not shown herein, computing device 150 may includeadditional engines, and detection engine 170 may comprise a number ofsub-engines. In examples described herein, any engine(s) of computingdevice 150 (e.g., engine 170) may be any combination of hardware andprogramming to implement the functionalities of the respective engine.Such combinations of hardware and programming may be implemented in anumber of different ways. For example, the programming may be processorexecutable instructions stored on a non-transitory machine-readablestorage medium (e.g., computer-readable medium 320) and the hardware mayinclude a processing resource to execute those instructions. In suchexamples, the machine-readable storage medium may store instructionsthat, when executed by the processing resource, implement the engines.The machine-readable storage medium storing the instructions may beintegrated in the same computing device (e.g., device 150) as theprocessing resource to execute the instructions, or the machine-readablestorage medium may be separate from but accessible to the computingdevice and the processing resource. The processing resource may compriseone processor or multiple processors included in a single computingdevice or distributed across multiple computing devices.

In some examples, the instructions can be part of an installationpackage that, when installed, can be executed by the processing resourceto implement the engines of system 100. In such examples, themachine-readable storage medium may be a portable medium, such as acompact disc, DVD, or flash drive, or a memory maintained by a serverfrom which the installation package can be downloaded and installed. Inother examples, the instructions may be part of an application orapplications already installed on a computing device including theprocessing resource (e.g., device 150). In such examples, themachine-readable storage medium may include memory such as a hard drive,solid state drive, or the like.

As used herein, a “machine-readable storage medium” may be anyelectronic, magnetic, optical, or other physical storage apparatus tocontain or store information such as executable instructions, data, andthe like. For example, any machine-readable storage medium describedherein may be any of a storage drive (e.g., a hard drive), flash memory,Random Access Memory (RAM), any type of storage disc (e.g., a compactdisc, a DVD, etc.), and the like, or a combination thereof. Further, anymachine-readable storage medium described herein may be non-transitory.

Referring to FIG. 8 in conjunction with FIGS. 7A and 7B, detectionengine 170 may obtain one or more images representing an object (e.g.,one or more RGB images representing the object, one or more infraredimages representing the object, and/or one or more depth imagesrepresenting the object) from one or more cameras of sensor bundle 164.In some examples, engine 170 may obtain at least two images representingthe object, and in some examples, the two images may be obtained fromtwo different cameras of sensor bundle 164. In some example, the twocameras may be any two cameras of sensor bundle 164.

Based on the obtained image(s), engine 170 may determine athree-dimensional location (e.g., coordinates) of the object (e.g., ofone or more fingers 37 a-37 j illustrated in FIGS. 7A and 7B). Thethree-dimensional coordinates may be expressed, for example, inCartesian coordinates (x, y, z), having a point of origin, for example,at the far-left corner of sensitive region 202, from a user'sperspective. In some examples, the “z” axis may be perpendicular tosurface 200 and the “y” axis may be parallel to centerline 205. In otherexamples, other coordinate systems and orientations may be used todefine the three-dimensional coordinates of the object(s).

Based on the obtained three-dimensional coordinates of an object, engine170 may determine two-dimensional coordinates of the object's projectiononto touch-sensitive surface 200. In some examples, the object'sprojection may be a parallel projection, where the object'sthree-dimensional coordinates are projected onto a point on surface 200via a line perpendicular to surface 200. For example, if thethree-dimensional coordinates of fingertip 37 j are (x1, y1, z1), itstwo-dimensional projection coordinates may be (x1, y1), as illustratedin FIG. 7B. In other examples, the objects projection may be aperspective projection, where the object's three-dimensional coordinatesmay be projected onto a point on surface 200 via a line that is notnecessarily perpendicular to surface 200, such as a line connecting theobject with a predefined point, e.g., some point on sensor bundle 164 orprojector assembly 184. Irrespective to what type of projection ischosen, engine 170 may obtain the two-dimensional coordinates of theobject projection onto surface 200 based on the objectsthree-dimensional coordinates and the location and orientation ofsurface 200.

In some examples, engine 170 may obtain the two-dimensional projectioncoordinates directly from the one or more images obtained from one ormore cameras of sensor bundle 164, without first obtaining thethree-dimensional coordinates of the object. For example, engine 170 mayuse one or more pre-calibrated transformation matrices to transform anobjects (two-dimensional) coordinates within the obtained image(s) intotwo-dimensional projection coordinates.

In some examples, based on the determined two-dimensional projectioncoordinates, engine 170 may determine two-dimensional displaycoordinates on display 152. The display coordinates may be associatedwith (or correspond to) the projection coordinates in various ways. Insome examples, display coordinates may be a linear or a non-linearfunction of the projection coordinates. For example, engine 170 may useone or more linear transformation matrices to transform the projectioncoordinates into the display coordinates. In some examples, thecoordinates are transformed such that point (0,0) in projectioncoordinates (e.g., the far-left corner of region 202) is transformedinto point (0,0) in display coordinates (e.g., top-left corner ofdisplay 152). Similarly, engine 170 may scale the “x” coordinate ofprojection coordinates by a ratio between the width of display 152 andthe width of region 202, and scale the “y” coordinate of the projectioncoordinates by a ratio between the height of display 152 and the heightof region 202. As a result, the near-right corner of region 202 may betransformed into the bottom-right corner of display 152.

After determining the display coordinates, engine 170 may display anobject indicator at the display coordinates on display 152. As usedherein, “displaying an object indicator” may refer, for example, toproviding the object indicator to display 152 for displaying the objectindicator, causing display 152 to display the object indicator. Forexample, as illustrated in FIG. 7B, engine 170 may determine projectedcoordinates of fingertips 37 a-37 j onto surface 200, as describedabove. Based on the projected coordinates, engine 170 may calculatedisplay coordinates, and display object indicators 156 a-156 j at ornear the respective calculated display coordinates. For example, forfingertip 37 j, engine 170 may determine projected coordinates (x1, y1).Based on these projected coordinates, engine 170 may calculate displaycoordinates (x2, y2), and display object indicator 156 j at thesecoordinates, as depicted in FIG. 7B.

The displayed object indicators may include any types of shapes, icons,graphics, etc., and may have different colors and different levels oftransparency or opacity. In some examples, all object indicatorsdisplayed on display 152 may have the same appearance, while in otherexamples, two simultaneously displayed indicators corresponding to twodifferent objects may be different (e.g., have a different appearance).In some examples, engine 170 may detect the object's type (e.g., afingertip, a stylus, etc.) and select one of a plurality of (e.g., atleast two) different object indicators based on the object's type.Engine 170 may further distinguish between objects of the same type. Forexample, engine 170 may detect, for each fingertip, to which finger onwhich hand that fingertip corresponds, and select different objectindicators for different fingers and hands.

In some examples, the one or more object indicators displayed on display152 may be combined with (e.g., overlaid upon) another image displayedon display 152. The other image may be provided to display 152 by anoperating system or an application being executed computing device 150.Accordingly, the user may use objects, such as fingertips, to identifyand select target areas of an image displayed on display 152, and theobject indicators can serve as visual guides or cursors indicating thelocations on display 152 corresponding to the objects' projection ontosurface 200.

As described above, in some examples, the objects (e.g., fingertips 37)may be disposed anywhere in the region between sensor bundle 164 andsurface 200. Accordingly, the objects may be touching surface 200,hovering above surface 200, etc. In some examples, engine 170 maydetermine (e.g., based on signals from surface 200) whether a particularobject is touching surface 200, and select a different object indicatorfrom a plurality of indicators (or modify the appearance of the sameobject indicator) based on the determination. For example,touch-sensitive surface 200 may detect one or more touches by objects,and pass touch data describing the one or more touches to engine 170.

Based on the touch data received from surface 200, engine 170 maydetermine, for each touch included in the touch data, whether the touchis associated with one of the objects (e.g., fingertips) detected, asdescribed above, based on image(s) from sensor bundle 164. For example,engine 170 may determine whether the location of a particular touch onsurface 200 coincides or is within a predefined distance (e.g., 1 mm)from the one of the objects or from one of the objects' projections ontosurface 200. For example, engine 170 may determine whether the locationof a particular touch is directly or substantially underneath of aparticular object, which may be determined, for example, based on thetouch data and the one or more images from camera(s) of sensor bundle164 (or the objects three-dimensional coordinates determined based onthe images). In some examples, engine 170 may determine whether aparticular touch is associated with a particular object by determiningthe object's distance from surface 200, for example, using depth datareceived from depth camera 164 c, or using the three-dimensionalcoordinates determined as described above. For example, if only one of aplurality of objects is touching surface 200, engine 170 may determinewhich object is touching surface 200 by determining the object with theminimal distance or with a distance below a predefined distance (e.g., 1mm).

In the example as illustrated in FIGS. 7A and 7B, fingertips 37 c and 37d are touching surface 200 while other fingertips are hovering abovesurface 200. In this example, engine 170 may detect two touches based ontouch data provided by surface 200 and determine (e.g., based on theimage(s) of fingertips 37 a-37 j or the three-dimensional locations offingertips 37 a-37 j determined based on the images) that the twotouches correspond to fingertips 37 c and 37 d. Accordingly, asillustrated in FIG. 7B, engine 170 may select object indicators 156 cand 156 d from a first type (e.g., a highlighted circle) and selectobject indicators 156 a, 156 b, and 156 e-156 j from a second(different) type (e.g., a non-highlighted circle). This may provide theuser with a confirmation that one or more touches have been detected,and also provide the user with a visual guidance as to which of theobject indicators displayed on display 152 correspond to touchingobjects and which correspond to non-touching (hovering) objects. Thismay allow the user to focus attention, for example, only on the objectsthat are touching surface 200. In some examples, engine 170 may beconfigured (e.g., by the user) to display object indicators only ofobjects that are touching surface 200, only of objects that are nottouching surface 200, or of both types of objects.

In some examples, engine 170 may determine whether or not to display anobject indicator of an object based on the object's location. Forexample, display 152 may be a touch-sensitive display, in which case theuser may sometimes choose to provide user input by using (e.g.,touching) display 152, rather than using the object detectionfunctionality described above. In such examples, engine 170 maydetermine that the object (e.g., the fingertip) moves closer to display152 and farther from surface 200, and based on that determination stopdisplaying the object indicator for that object or for all objects.Similarly, engine 170 may determine that the object moves farther fromdisplay 152 and closer to surface 200, and based on that determination,start displaying the object indicator for that object.

In particular, in some examples engine 170 may determine whether or notto display an object indicator of an object based on the object'sdistance from surface 200, based on the object's distance from display152, or based on both distances. For example, engine 170 may display anobject indicator of an object if the object is within a predefineddistance (e.g., 150 mm) from surface 200, and not display (or stopdisplaying) the object indicator if the object is not within (or movesoutside of) the predefined distance from surface 200. As anotherexamples, engine 170 may display an object indicator of an object if theobject is within a predefined distance of display 152, and not display(or stop displaying) the object indicator if the object is not within(or moves outside of) the predefined distance from display 152. In someexamples, engine 170 may display an object indicator of an object if theobject's distance from surface 200 is smaller than the objects distancefrom display 152 and not display the object indicator otherwise.

In some examples, engine 170 may determine whether or not to display anobject indicator of an object based on a ratio or difference between theobject's distance from surface 200 and the object's distance fromdisplay 152. For example, engine 170 may display an object indicator ofan object if the ratio or difference is smaller than a predefined ratioor predefined difference and not display (or stop displaying) the objectindicator if the ratio or difference is greater or equal to thepredefined ratio or predefined difference. In some examples, engine 170may determine whether or not to display an object indicator of an objectbased on a distance between the object and sensor bundle 164, asdetermined, for example, based on input from ambient light sensor 164Dor other camera(s) in sensor bundle 164. For example, engine 170 maydisplay an object indicator of an object if the distance is smaller thana predefined distance, and not display (or stop displaying) the objectindicator if the distance is greater or equal to the predefineddistance.

In some examples, engine 170 may implement a hysteresis mechanism andadjust the predefined distances, ratios, differences, and otherthresholds discussed above when the object crosses the threshold. Thismay prevent jittering (rapid displaying and hiding of the objectindicator) when the object is located near the threshold, and isunintentionally crossing it back and forth. In some examples, thevarious thresholds described above may be predefined and fixed, orconfigurable by the user. In some examples, the object's distances fromdisplay 152 and surface 200 may be determined by engine 170, forexample, based on the three-dimensional coordinates of the object, asdescribed above, or using other suitable methods, such as using depthdata received from depth camera 164 c or other cameras of sensor bundle164. In some examples, engine 170 may determine for each object whetherto display an object indicator for that object. Thus, in some examples,engine 170 may simultaneously display object indicators for one or moredetected objects, and not display object indicators for other detectedobject(s). In other examples, engine 170 may not display objectindicators for any detected object if engine 170 determines that atleast one object indicator should not be displayed, using any of thetechniques described above.

While in some of the above examples engine 170 is described as beingconfigured to determine the object's location and to display acorresponding object indicator only once, it is appreciated that engine170 may be configured to perform the described functionalitycontinuously, for example, a fixed number of times (e.g., 30) persecond. Thus, in some examples, as the one or more objects are movingand as some objects are touching or stopping touching surface 200,engine 170 may continuously and in real time detect new object locationsand touches, and update the displayed object indicators accordingly.

FIG. 9 is a block diagram of an example portion of computing device 150.In the example of FIG. 9, computing device 150 is communicativelyconnected to touch-sensitive surface 200, cameras 164A-164C, and display152, as described above. Each of cameras 164A-164C may be disposed aboveand pointed at surface 200. Computing device 150 may further include aprocessing resource 310 and a machine-readable storage medium 320comprising (e.g., encoded with) instructions 322-324.

In some examples, storage medium 320 may include additionalinstructions. In some examples, instructions 322-324 and any otherinstructions described herein in relation to storage medium 320 may bestored on a machine-readable storage medium remote from but accessibleto computing device 150 and processing resource 310. Processing resource310 may fetch, decode, and execute instructions stored on storage medium320 to implement the functionalities described herein. In otherexamples, the functionalities of any of the instructions of storagemedium 320 may be implemented in the form of electronic circuitry, inthe form of executable instructions encoded on a machine-readablestorage medium, or a combination thereof. Machine-readable storagemedium 320 may be a non-transitory machine-readable storage medium.

In some examples, instructions 322 may determine three-dimensionalcoordinates of an object, for example, based on at least two imagesobtained from two different cameras of sensor bundle 164, as describedabove. In some examples, as described above, the object may be disposedon or above a surface (e.g., hovering above or touching the surface),where the surface may be any surface other than the surface of adisplay, e.g., any surface that is not parallel to the surface of thedisplay, such as touch-sensitive surface 200. Based on thethree-dimensional object coordinates, instructions 324 may determinetwo-dimensional display coordinates that may be associated with orcorrespond to two-dimensional projection coordinates corresponding tothe object's projection onto the surface, as described above.Instructions 324 may display an object indicator at the determineddisplay coordinates on the display.

As described above, in some examples the surface may be atouch-sensitive surface, and medium 320 may also include instructions toreceive from the touch-sensitive surface touch data representing atleast one touch; determine whether the touch is associated with theobject, as described above; and select the object indicator from atleast two different object indicators based on whether the touch isassociated with the object. Medium 320 may also include instructions todetect the type of the object (e.g., a stylus, a fingertip, etc.) andselect the object indicators based on the detected type, as describedabove. Further, medium 320 may include instructions to determine whetheror not to display the object indicator based on the object's distancefrom the surface and the object's distance from the display (e.g., basedon the ratio and/or the difference between the two distances).

In some examples, features and functionalities described herein inrelation to FIG. 9 may be provided in combination with features andfunctionalities described herein in relation to any of FIGS. 1-8 and 10.

FIG. 10 is a flowchart of an example method 900 for displaying at leastone object indicator. Method 900 may be performed, for example, by atleast one computing system (e.g., computing system 100) having at leastone computing device (e.g., computing device 150) having at least oneprocessing resource (e.g., processing resource 310), or by any othercombination of hardware and/or software processors, computing devicesand/or computing systems.

At block 905, method 900 may determine locations (e.g.,three-dimensional locations) of a plurality of objects, where at leastone object may be touching a surface (e.g., touch-sensitive surface 200)and at least one other object may not be touching the surface. Forexample, as described above, the method may determine the locationsbased on an image or a plurality of images obtained from one or morecameras of sensor bundle 164. At block 910, method 900 may determine,based at least on the locations, display coordinates for the object orfor each of the plurality of objects. As described above, the displaycoordinates of an object may correspond to (e.g., be a linear functionof) the objects projection onto the surface. The surface may be anysurface other that the surface of a display (e.g., display 152), such astouch-sensitive surface 200. At block 915, the method may display anobject indicator for one or each of the plurality of objects on adisplay (e.g., display 152), as described above.

In some examples, method 900 may include additional blocks. For example,method 900 may also detect one or more touches on the surface (e.g., ifthe surface is a touch-sensitive surface such as surface 200) andselect, for each object, an object indicator from at least two differentobject indicators based on whether the object corresponds to any of thetouches (e.g., touches the touch-sensitive surface). In some examples,the display may be a touch-sensitive display, and the method may alsodetermine whether to display an object indicator object for a particularobject based on that object's distance from the surface and its distancefrom the display (e.g., based on the ratio and/or the difference betweenthe two distances).

Although the flowchart of FIG. 10 shows a specific order of performanceof certain functionalities, method 900 is not limited to that order. Forexample, the functionalities shown in succession in the flowchart may beperformed in a different order, may be executed concurrently or withpartial concurrence, or a combination thereof. In some examples,features and functionalities described herein in relation to FIG. 10 maybe provided in combination with features and functionalities describedherein in relation to any of FIGS. 1-9.

What is claimed is:
 1. A computing system comprising: a touch-sensitivesurface; a display, wherein the display is not parallel to thetouch-sensitive surface; at camera to capture an image representing anobject disposed between the camera and the touch-sensitive surface; anda non-transitory machine-readable storage medium storing instructionsexecutable by a processing resource to: determine display coordinatescorresponding to the object's projection onto the touch-sensitivesurface; select an object indicator; determine to display the objectindicator based on a combination of both the object's distance from thetouch-sensitive surface and the object's distance from the display; andprovide the selected object indicator to the display for displaying theselected object indicator at the determined display coordinates on thedisplay.
 2. The computing system of claim 1, wherein the instructionsfurther comprise instructions to determine the display coordinatescorresponding to the object's projection based on the image representingthe object.
 3. The computing system of claim 1, wherein the instructionsfurther comprise instructions to select the object indicator from atleast two different object indicators based on whether the object istouching the touch-sensitive surface.
 4. The computing system of claim1, wherein the instructions further comprise instructions to determinewhether the touch detected by the touch-sensitive surface is associatedwith the object.
 5. The computing system of claim 4, wherein theinstructions further comprise instructions to determine whether thetouch detected by the touch-sensitive surface is associated with theobject based on whether a location of the touch coincides or is within apredefined distance from the object or from the objects' projections onthe touch sensitive surface.
 6. The method of claim 5, wherein thedetermination of the locations is based on a plurality of imagesobtained from at a RGB camera, an infrared camera, a depth camera, orcombinations thereof.
 7. The computing system of claim 1, wherein theinstructions further comprise instructions to determine to display theobject indicator based on a ratio of the object's distance from thetouch-sensitive surface and the object's distance from the display. 8.The computing system of claim 1, wherein the instructions furthercomprise instructions to determine whether to display the objectindicator based on a difference between the object's distance from thetouch-sensitive surface and the object's distance from the display.
 9. Amethod comprising: determining a locations of a plurality of objectsincluding an object that is not touching either of a touch-sensitivesurface or a display that is not parallel to the touch-sensitivesurface; determining for each object in the plurality of objects displaycoordinates corresponding to coordinates of each object's projectiononto the touch-sensitive surface; determine to display an objectindicator for the object not touching either of the touch-sensitivesurface or display based on a combination of both the object's distancefrom the touch-sensitive surface and the object's distance from thedisplay; and display the determined object indicator at the determineddisplay coordinates on the display.
 10. A non-transitorymachine-readable storage medium comprising instructions executable by aprocessing resource to: determine three-dimensional coordinates of anobject; based on the three-dimensional coordinates of the object,determine two-dimensional display coordinates corresponding totwo-dimensional projection coordinates, wherein the two-dimensionalprojection coordinates correspond to the object's projection onto atouch-sensitive surface other than a display's surface included in acomputing system, wherein the display's surface is not parallel to thetouch-sensitive surface; determine to display an object indicator basedon a combination of both the object's distance from the touch-sensitivesurface and the object's distance from the display; and provide fordisplay of the object indicator at the determined display coordinates onthe display.
 11. The non-transitory machine-readable storage medium ofclaim 10, wherein the instructions further comprising instructions todetermine the three-dimensional coordinates of the object based on atleast two images obtained from two different cameras of a plurality ofcameras including an RGB camera, an infrared camera, and a depth camera.12. The non-transitory machine-readable storage medium of claim 10,wherein the instructions further comprise instructions to detect a typeof the object, and wherein selecting the object indicator is furtherbased on the type of the object.
 13. The non-transitory machine-readablestorage medium of claim 12, wherein the instructions further comprisinginstructions to detect whether the object is a stylus, a fingertip, agame piece, a book, a mug, a document, or a photo.
 14. Thenon-transitory machine-readable storage medium of claim 10, wherein theinstructions further comprising instructions to determine to display theobject indicator based on the object's distance from the touch-sensitivesurface and the object's distance from the display when the object isnot touching either of the touch-sensitive surface or the display.